The present invention relates to the development of tests to meet test specifications. More specifically, but not exclusively, the present invention relates to methods and systems for the selection of items for inclusion in a test in order to meet test specifications.
A test specification is a set of content and statistical requirements designed to ensure the validity, reliability, and fairness of test scores. Regardless of whether the test is a computer-adaptive test (CAT) or a traditional pencil and paper test, there are a number of key concerns for developing a test to meet the test specifications. With respect to the selection of items (questions) to administer, key concerns include test content, test reliability, test validity, test security, and test item usage. Test content relates to the subject matter and skill coverage of the test. Test validity relates to the appropriateness of the score interpretations proposed for the test. Test reliability relates to the accuracy of the test or the consistency of scores. Test security relates to the likelihood that an examinee taking a particular test would have previously seen test items. Test item usage relates to the proportion of available test items actually administered to examinees and to the frequency with which an item is administered to examinees.
Test specifications include a set of content and statistical requirements designed to ensure the validity, reliability, and fairness of test scores. Typical prior art attempts at meeting the specifications for a test include application of complex, stochastic processes that treat content, statistical, and item usage concerns simultaneously through a method known as the weighted deviations model. Because the weighted deviations model is a stochastic process, it only assures that test statistical, content, and security requirements are met in probability, which is to say on average. Thus, a likelihood remains that individual test takers may receive tests that do not conform to either test content specifications or test statistical specifications. Furthermore, because the weighted deviations model, as typically implemented, confounds content and information in the selection of items, and information is not distributed evenly over content categories, highly discriminating items become over-exposed without any concomitant benefit in measurement precision. Therefore, there are problems with typical prior art.
Therefore, it is a primary object, feature, or advantage of the present invention to improve upon the state of the art.
It is another object, feature, or advantage of the present invention to provide a method and system for selecting items in a manner that meets key goals of test assembly including test content, test validity, test reliability, test security, and test item usage.
Yet another object, feature, or advantage of the present invention is to meet the key goals of test assembly by separating the process for meeting test content from that for meeting test statistical and item usage specifications.
A further object, feature, or advantage of the present invention is to eliminate the stochastic elements of the test assembly process and make the content representation deterministic.
A still further object, feature, or advantage of the present invention is to ensure that all test takers receive the correct content representation.
Another object, feature, or advantage of the present invention is to provide for a deterministic content representation such that it becomes possible to make extremely accurate item development needs forecasts.
Yet another object, feature, or advantage of the present invention is to replace the weighted deviations model by using an item selection process that seeks to equalize usage over items without compromising the precision with which scores are measured.
A further object, feature, or advantage of the present invention is to insure that all items within an item collection associated with a slot are available for selection some portion of the time.
A still further object, feature, or advantage of the present invention is to increase the number of items within an item pool actually being administered and reduce the usage for any given item.
Another object, feature, or advantage of the present invention is to provide a method and system for selecting items for inclusion in a test that allows for assembly of either linear or adaptive computer-based tests.
One or more of these and/or other objects, features, or advantages of the present invention will become apparent from the specification and claims that follow.